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^ Abstract 

^ This paper develops new theory and algorithms to recover signals that are 

_^ approximately sparse in some general (i.e., basis, frame, over-complete, or in- 
complete) dictionary but corrupted by a combination of measurement noise 

L_( and interference having a sparse representation in a second general dictionary. 

1^ Particular applications covered by our framework include the restoration of 

^ signals impaired by impulse noise, narrowband interference, or saturation, 

'— ' as well as image in-painting, super-resolution, and signal separation. We de- 

,_H velop efficient recovery algorithms and deterministic conditions that guaran- 

^ tee stable restoration and separation. Two application examples demonstrate 

^ the efficacy of our approach. 

o 
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1. Introduction 



*k>( We investigate the problem of recovery of the coefficient vector x G C^" 

?H from the corrupted M-dimensional observations 

z = Ax + Be + n, (1) 

where A G C^^^" and B G C'^^^'' are general (basis, frame, over-complete, 
or in-complete) deterministic dictionaries, i.e., matrices whose columns have 
unit Euclidean (or £2) norm. The vector x is assumed to be approximately 
sparse, i.e., its main energy (in terms of the sum of absolute values) is con- 
centrated in only a few entries. The M-dimensional signal vector is defined 
as y = Ax. The vector e G C^"" represents interference and is assumed to be 
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perfectly sparse, i.e., only a few entries are nonzero, and n G C^^ corresponds 
to measurement noise. Apart from the bound ||n||2 < e, the measurement 
noise is allowed to be arbitrary. The interference and noise components e 
and n are allowed to depend on the vector x and/or the dictionary A. 

The setting ([I| also allows us to study signal separation, i.e., the separa- 
tion of two distinct features Ax and Be from the noisy observation z. Here, 
the vector e in ([I| is also allowed to be approximately sparse and is used 
to represent a second desirable feature (rather than undesired interference). 
Signal separation amounts to simultaneously recovering the vectors x and e 
from the noisy measurement z followed by computation of the individual 
signal features Ax and Be. 



1.1. Applications for the model (lly 

Both the recovery and separation problems outlined above feature promi- 
nently in numerous applications (see [THT7] and references therein), including: 

• Impulse noise: The recovery of approximately sparse signals corrupted 
by impulse noise [13j corresponds to recovery of x from ([I]) by setting 
B = Im and associating the interference e with the impulse-noise vec- 
tor. Practical examples include restoration of audio signals impaired 
by click/pop noise [U E] and reading out from unreliable memory jT] 



• 



• 



Narrowband interference: Audio, video, or communication signals are 
often corrupted by narrowband interference. A particular example 
is electric hum, which typically occurs in improperly designed audio 
equipment. Such impairments naturally exhibit a sparse representation 
in the frequency domain, which amounts to setting B to the Fourier 
matrix. 

Saturation: Non-linearities in amplifiers may result in signal saturation, 
e.g., [71 Uni [H]. Here, instead of the signal vector y of interest, one 
observes a saturated (or clipped) version z = y + e + n, where the 
nonzero entries of e correspond to the difference between the saturated 
signal and the original signal y . The noise vector n can be used to model 
residual errors that are not captured by the interference component Be. 

Super-resolution and in-painting: In super-resolution jHl [I5] and in- 
painting [HI l8]-[l0] applications, only a subset of the entries of the (full- 
resolution) signal vector y = Ax is available. With ([I|, the interference 



vector e accounts for the missing parts of the signal, i.e., the locations 
of the nonzero entries of e correspond to the missing entries in y and 
are set to some arbitrary value. The missing parts of y are then filled 
in by recovering x from z = Ax + e + n followed by computation of 
the (full-resolution) signal vector y = Ax. 

• Signal separation: The framework ([I| can be used to model the de- 
composition of signals into two distinct features. Prominent applica- 
tion examples are the separation of texture from cartoon parts in im- 
ages lU E] and separation of neuronal calcium transients from smooth 
signals caused by astrocytes in calcium imaging [5]. In both applica- 
tions, A and B are chosen such that each feature can be represented 
by approximately sparse vectors in one dictionary. Signal separation 
then amounts to simultaneously extracting x and e from z, where Ax 
and Be represent the individual features. 

In almost all of the applications outlined above, a predetermined (and 
possibly optimized) dictionary pair A and B is used. It is therefore of signif- 
icant practical interest to identify the fundamental limits on the performance 
of restoration or separation from the model ([I| for the deterministic setting, 
i.e., assuming no randomness in the dictionaries, the signal, interference, or 
the noise vector. Deterministic recovery guarantees for the special case of 
perfectly sparse vectors x and e and no measurement noise have been stud- 
ied in [T2I [T8] . The results in [T2l [T8] rely on an uncertainty relation for 
pairs of general dictionaries and depend on the number of nonzero entries 
of X and e, on the coherence parameters of the dictionaries A and B, and 
on the amount of prior knowledge on the support of the signal and interfer- 
ence vector. However, the algorithms and proof techniques used in fTIi [TH] 
cannot be adapted for the general (and practically more relevant) setting 
formulated in ([I|, which features approximately sparse signals and additive 
measurement noise. 

1.2. Contributions 

In this paper, we generalize the recovery guarantees of [T^ [TS] to the 
framework ([I| detailed above. In particular, we provide novel, computation- 
ally efficient restoration and separation algorithms, and derive corresponding 
recovery guarantees for the deterministic setting. Our guarantees depend in 
a natural way upon the number of dominant nonzero entries of x and e, on 



the coherence parameters of the dictionaries A and B, and on the Euchdean 
norm of the measurement noise. Our results also depend on the amount of 
knowledge on the location of the dominant entries available prior to recov- 
ery. In particular, we investigate the following cases: 1) The locations of 
the dominant entries of the approximately sparse vector x and the support 
set of the perfectly sparse interference vector e are known (prior to recov- 
ery), 2) only the support set of the interference vector e is known, and 3) no 
support-set knowledge about x and e is available. Moreover, we present new 
coherence-based bounds on the restricted isometry constants (RICs) for the 
cases 2) and 3), which we then use to derive alternative recovery conditions 
using the restricted isometry property (RIP) framework. We provide a com- 
parison to the recovery conditions for perfectly sparse signals and noiseless 
measurements presented in p^ 118]. Finally, we demonstrate the efficacy of 
our approach with two representative applications: Restoration of audio sig- 
nals that are impaired by a mixture of impulse noise and Gaussian noise, and 
removal of scratches from old photographs. 

1.3. Notation 

Lowercase and uppercase boldface letters stand for column vectors and 
matrices, respectively. The transpose, conjugate transpose, and (Moore- 
Penrose) pseudo-inverse of the matrix M are denoted by M^, M^, and 
M"!^ = (M'^M) M^, respectively. The kth entry of the vector m is [m]fc, 
and the A;th column of M is rrifc and the entry in the kth row and ith col- 
umn is designated by [M]^^^. The M x M identity matrix is denoted by 
Im and the M x iV all zeros matrix by OmxA^- The Euclidean (or £2) norm 
of the vector x is denoted by ||x||2, ||x||-^ = X]fc|[x]fc| stands for the £i-norm 
of x, and ||x||q designates the number of nonzero entries of x. The spec- 
tral norm of the matrix M is ||M||2 = A/Amax(M-^M), where the minimum 
and maximum eigenvalue of a positive-semidefinite matrix M are denoted 

by Amin(M) and Amax(M), respectively. ||M||^ = ^J^k,i\[^]k,ef stands for 
the Frobenius matrix norm. Sets are designated by upper-case calligraphic 
letters. The cardinality of the set T is \T\ and the complement of a set S 
in some superset T is denoted by S'^. The support set of the vector m, i.e., 
the index set corresponding to the nonzero entries of m, is designated by 
supp(m). We define the M x M diagonal (projection) matrix P5 for the set 



5 C {1,...,M} as follows: 

r -. _ / 1, k = i and k E S 
[ s\k,i = j 0, otherwise, 

and 1117- = P7-m. The matrix M7- is obtained from M by retaining the 
columns of M with indices in T and the |T| -dimensional vector [m]7- is 
obtained analogously. For a; G M, we set [x] = max{x, 0}. 

1.4. Outline of the paper 

The remainder of the paper is organized as follows. In Section [2| we 
briefly summarize the relevant prior art. Our new recovery algorithms and 
corresponding recovery guarantees are presented in Section |3j An alterna- 
tive set of recovery guarantees obtained through the RIP framework and a 
comparison to existing recovery guarantees are provided in Section |4j The 
application examples are shown in Section |5j and we conclude in Section |6| 

2. Relevant Prior Art 

In this section, we review the relevant prior art in recovering sparse sig- 
nals from noiseless and noisy measurements in the deterministic setting and 
summarize the existing guarantees for recovery of sparsely corrupted signals. 

2.1. Recovery of perfectly sparse signals from noiseless measurements 

Recovery of a vector x G C^" from the noiseless observations y = Ax 
with A over-complete (i.e., M < Na) corresponds to solving an underde- 
termined system of linear equations, which is well-known to be ill-posed. 
However, assuming that x is perfectly sparse (i.e., that only small number of 
its entries are nonzero) enables us to uniquely recover x by solving 

(PO) minimize ||x||q subject to y = Ax. 

Unfortunately, PO has a prohibitive (combinatorial) computational complex- 
ity, even for small dimensions A^^. One of the most popular and computation- 
ally tractable alternative to solving PO is basis pursuit (BP) [T91 - 124J . which 
corresponds to the convex program 

(BP) minimize \\^\\i subject to y = Ax. 



Recovery guarantees for PO and BP are usually expressed in terms of the 
sparsity level 71^ = ||x||q and the coherence parameter of the dictionary A, 
which is defined as 

Ua = max laFa^l . 

Specifically, a sufficient condition for x to be the unique solution of PO and 
for BP to deliver this solutioiQis [271 [221 [2l] 

2.2. Recovery of approximately sparse signals from noisy measurements 

For the case of bounded (otherwise arbitrary) measurement noise, i.e., 
z = Ax + n with ||n||2 < e, recovery guarantees based on the coherence 
parameter fia were developed in J271 - I3T] . The corresponding recovery con- 
ditions mostly treat the case of perfectly-sparse signals, i.e., where only a 
small fraction of the entries x are nonzero. In almost all practical applica- 
tions, however, only a few entries of x are actually zero. Nevertheless, many 
real-world signals exhibit the property that most of the signal's energy (in 
terms of the sum of absolute values) is concentrated in only a few entries. 
We refer to this class of signals as approximately sparse in the remainder of 
the paper. For such signals, the support set associated to the best n^-sparse 
approximation (in £i-norm) corresponds to 

X = supp„Jx) = arj min ||x - x^^H^ , 

where the set S„^ contains all support sets of size n^ corresponding to per- 
fectly n^j-sparse vectors having the same dimension as x. A particular sub- 
class of approximately sparse signals is the set of compressible signals, whose 
approximation error decreases according to a power law with exponent s > 
as Hx increases [32]. 

The following theorem provides a sufficient condition for which a suitably 
modified version of BP, known as BP denoising (BPDN) |_19j, stably recovers 
an approximately sparse vector x from the noisy observation z. 



^The condition ([2| also ensures perfect recovery using orthogonal matching pur- 
suit (OMP) |24ff26] . which is, however, not further investigated in this paper. 
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Theorem 1 (BP denoising [31, Thm. 2.1]). Let z = Ax + n, ||n||2 < e, 

and X = supp^ (x) . // M) is met, then the solution x of the convex program 



(BPDN) minimze ||x||]^ subject to ||z — Ax||2 < rj 

with 6 < rj satisfies 

||x-x||2 < Co{e + T]) + Ci||x-xa'||i, (3) 

where both (non-negative) constants Cq and Ci depend on fia andux- 

Proof. The proof in |3T| Thm. 2.1] is detailed for perfectly sparse vectors 
only. Since some of the proofs presented in the remainder of the paper are for 
approximately sparse signals and noisy measurements, we present the general 
Appendix A 



case m 



We emphasize that perfect recovery of x is, in general, impossible in the 
presence of bounded (but otherwise arbitrary) measurement noise n. In the 
remainder of the paper, we consider stable recovery instead, i.e., in a sense 
that the £2-iiorm of the difference between the estimate x and the ground 
truth X is bounded from above by the ^2-iiorm of the noise ||n||2 and the 
best na;-sparse approximation in £i-norm sense, i.e., as in d3|. We finally 
note that Theorem [T] generalizes the results for noiseless measurements and 



perfectly sparse signals in |2T | l22 | [M] using BP (cf. Section 2.1). Specifically, 
for ||n||2 = and ||x — X;t'||j^ = 0, BPDN with ?] = corresponds to BP 
and (jsl) results in ||x — x||2 = 0, which ensures perfect recovery of the vector x 
whenever ^ is met. 

2.3. Recovery guarantees for perfectly sparse signals from sparsely corrupted 
and noiseless measurements 

A large number of restoration and separation problems occurring in prac- 
tice can be formulated as sparse signal recovery from sparsely corrupted 
signals using the input-output relation (jTl). Special cases of the general 
model were studied in |7l MM, M, EM?]. 

Probabilistic recovery guarantees. Recovery guarantees for the probabilistic 
setting (i.e., recovery of x is guaranteed with high probability) for random 
Gaussian matrices, which are of particular interest for applications based on 
compressive sensing (CS), were reported in j7l [HI [Ml |37j. Similar results 



for randomly sub-sampled unitary matrices A were developed in |3S]- The 
problem of sparse signal recovery from a particular nonlinear measurement 
process in the presence of impulse noise was considered in [13], and prob- 
abilistic results for signal detection based on £i-norm minimization in the 
presence of impulse noise was investigated in [34j . In the remainder of the 
paper, however, we will consider the deterministic setting exclusively. 

Deterministic recovery guarantees. Recovery guarantees in the deterministic 
setting for noiseless measurements and signals being perfectly sparse, i.e., 
the model z = Ax + Be, were studied in |T^ [T5| [55] . In j33], it was shown 
that when A is the Fourier matrix, B = Im and when the support set of 
the interference e is known, perfect recovery of x is possible if 2nxne < M, 
where n^ = \\g\\q. The case of A and B being arbitrary dictionaries (whereas 
X and e are assumed to be perfectly sparse and for noiseless measurements) 
was studied for different cases of support-set knowledge in [T^ ITS] . There, 
deterministic recovery guarantees were presented, which depended upon the 
number of nonzero entries n^ and rie in x and e, respectively, and on the co- 
herence parameters /la and /x^ of A and B, as well as on the mutual coherence 
between the dictionaries A and B, which is defined as 

Urn = max laf^b^l . 

k,l 

A summary of the recovery guarantees presented in [T2l [18] (along with the 
novel recovery guarantees presented in the next section) is given in Table [11 
where, for the sake of simplicity of exposition, we define the function 

/(n, v) = [l- fiaiu - 1)]+ [1 - fibiv - 1)]+ . 

We emphasize that the results presented in [T2l [TSj are for perfectly sparse 
and noiseless measurements only, and furthermore, the algorithms and proof 
techniques cannot be adapted for the more general setting proposed in ([I]). In 
order to gain insight into the practically more relevant case of approximately 
sparse signals and noisy measurements, we next develop new restoration and 
separation algorithms for several different cases of support-set knowledge and 
provide corresponding recovery guarantees. Our results complement those in 
[121 [H] (cf. Table [l|. 

3. Main Results 

We now develop several computationally efficient methods for restoration 
or separation under the model M and derive corresponding recovery condi- 



Table 1: Summary of deterministic recovery guarantees for perfectly/ approximately sparse 
signals that are corrupted by interference in the absence/presence of measurement noise. 



Support-set 
knowledge 


Recovery condition 


Perfectly sparse 
and no noise 


Approx. sparse 
and noise 


X and e 


^a-^e/im</(nx,ne) 


12, Thm. 3 


Theorem 


2 




e only 


2n^nei^l,<fi2n^,ne) 


[12l Thnis. 4 and 5] 


Theorem 


3 




None 


PlEq 
Eq. 


. 12]^ 
13 


[TKl Thm. 3 


Theorem 4 



■^The recovery condition is valid for BP and OMP; a less restrictive condition for PO is 
given in fW. Thm. 2]. 



tions that guarantee their stability. Our recovery guarantees depend on the 
£2-norm of the noise vector and on the amount of knowledge on the domi- 
nant nonzero entries of the signal and noise vectors. Specifically, we consider 
the following three cases: 1) Direct restoration: The locations of the entries 
corresponding to the best n^-sparse approximation of x and the support set 
of the (perfectly sparse) interference vector e are known prior to recovery, 2) 
BP restoration: Only the support set of e is known, 3) BP separation: No 
knowledge about x and e is available, except for the fact that each vector 
exhibits an approximately an sparse representation in either A or B. 

3.1. Direct restoration: Support-set knowledge of ^ and e 

We start by addressing the case where the locations of the dominant 
entries (in terms of absolute value) of the approximately sparse vector x and 
the support set £ associated with the perfectly sparse interference vector e 
are known prior to recovery. This scenario is relevant, for example, in the 
restoration of old phonograph records [H [2], where one wants to recover a 
bandlimited signal that is impaired by impulse noise, e.g., clicks and pops. 
The occupied frequency band of phonograph recordings is typically known 
prior to recovery. In this case, one may assume A to be the M-dimensional 
discrete cosine transform (DOT) matrix and B = Im- The locations of the 
clicks and pops, i.e., the support set £ = supp(e), can be determined (prior 
to recovery) using the techniques described in [2], for example. 

The restoration approach considered for this setup is as follows. Since £ 
and X = supp,„Jx) are both known prior to recovery, we start by projecting 
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the noisy observation vector z onto the orthogonal complement of the range 
space spanned by B^:, which eliminates the sparse corruptions caused by the 
interference e. In particular, we consider 

R^-z = R£-(Ax + Be£- + n) = RfAx + Rfii, (4) 

where R^- = 1^/ — B^-B^ is the projector onto the orthogonal complement of 
the range space of B^:, and we used the fact that R£:Be£- = Oa/xi- Next, we 
separate Q by exploiting the fact that X is known 

RfZ = R£:A(xa' + Xa-O + R£-n 

= R£:A;t[x]A' + R^Ax;t- + Rfn 

and isolate the dominant entries ['x\x as follows: 

(R^Ax)^R^z = [x];t + (RsAxyRsiA^xc + n). (5) 

In the case where both vectors X;t'c and n are equal to zero, we obtain 

(Rf Ax)^R^z = [x];t, (6) 

and therefore the dominant entries of x are recovered perfectly by this ap- 
proach. Note that conjugate gradient methods (see, e.g., |3H]) offer an effi- 
cient way of computing ([g]) . 

The following theorem provides a sufficient condition for (R£:Ax)^R£ 
to exist and for which the vector x can be restored stably from the noisy 
measurement z using the direct restoration (DR) procedure outlined above. 

Theorem 2 (Direct restoration). Let z = Ax + Be + n with ||n||2 < e, 
e perfectly rif^-sparse with support set S, and X = supp.^J(x). Furthermore, 
assume that £ and X are known prior to recovery. If 

n^ne/i^ < /(n^,ne), (7) 

then the vector x computed according to 

(DR) [X];f = (R£A;,)tR^Z, [X]^c = 0|;t1xl 

with Us = Im — ^e^£ satisfies 

||x -x||2 < C^e + Qllx - X;f 111 , 

where the (non-negative) constants C3 and C4 depend on the coherence pa- 
rameters fia, f^b, o-nd fim, and on the sparsity levels Ux and He- 
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Proof. The proof is given in Appendix B 



Theorem [2] and in particular ([T]) provides a sufficient condition on the 
number rix of dominant entries of x for which DR enables the stable recovery 
of X from z, given the coherence parameters /i^, /i^, and /i^, and the number 
of sparse corruptions n^,. Specifically, ([T]) states that for a given number of 
sparse corruptions rig, the smaller the coherence parameters yU^, yU^, and /Xm, 
the more dominant entries of x can be recovered stably from z. The case 
that guarantees the recovery of the largest number Ux of dominant entries in 
X is when A and B are orthonormal bases (ONBs) (i.e., fia = fJ'b = 0) that 
are maximally incoherent (i.e., fXrn = '^/vM)', this is the case for the Fourier- 
identity pair, whence the recovery condition ([T]) corresponds to n^-ne < M. 

The recovery guarantee in Theorem ^ generalizes that in fT2^, Thm. 3] 
to approximately sparse signals and noisy measurements. In particular, for 
||n||2 = and ||x — :six\\i = 0, we have e = and that DR perfectly recovers x 
if (It]) is met. Since ([T]) is identical to the condition |T2| Thm. 3] (cf. Table [I]) 
we see that considering approximately sparse signals and (bounded) mea- 
surement noise does not result in a more restrictive recovery condition. We 
finally note that the recovery condition in ([T]) was shown in [12j to be tight for 
certain comb signals in the case where A is the Fourier matrix and B = Im- 

3.2. BP restoration: Support-set knowledge of e only 

Next, we find conditions guaranteeing the stable recovery in the setting 
where the support set of the interference vector e is known prior to recov- 
ery. A prominent application for this setting is the restoration of saturated 
signals [71 [16]. Here, no knowledge on the locations of the dominant en- 
tries of X is required. The support set £ of the sparse interference vector 
can, however, be easily identified by comparing the measured signal entries 
[z]j, 2 = 1,... , M, to a saturation threshold. Further application examples 
for this setting include the removal of impulse noise jll [21 [H], in-painting, 
and super-resolution |3l [H [15] of signals admitting an approximately sparse 
representation in some arbitrary dictionary A. 

The recovery procedure for this case is as follows. Since 8 is know prior 
to recovery, we may recover the vector x by projecting the noisy observation 
vector z onto the orthogonal complement of the range space spanned by Bg 



(cf. Section 3.1). This projection eliminates the sparse noise and leaves us 



with a sparse signal recovery problem similar to that in Theorem [T| In 
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particular, we consider recovery from 

R£:z = R£:(Ax + Be£- + n) = R£:Ax + R£:n, (8) 

where R^- = Ijv/— B^-B^, and we used the fact that R^-Be^ = OAfxi- Note that 
the idea of projecting the observation vector onto the orthogonal complement 
of the space spanned by the active columns of B was put forward in [7j, where 
the special case of B being an ONB was considered. The following theorem 
provides a sufficient condition that guarantees the stable restoration of the 
vector X from 



Theorem 3 (BP restoration). Let z = Ax + Be + n with ||n||2 < e. 
Assume e to be perectly n^-sparse and S = supp{e) to be known prior to 
recovery. Furthermore, let X = supp^Jx.) . If 

2n^neiJ,l,< f{2n^,ne), (9) 

then the result x of BP restoration 

(BP-RES) I ™^«™^e ||i||^ 

^ [^ subject to ||R£-(z — Ax)||2 < r] 

with He = Im — B^-B^ and e < r] satisfies 

||x-x||2 < C^ie + T]) + C6||x-xa'||i, 

where the (non-negative) constants C5 and Cq depend on the coherence pa- 
rameters fia, f^b, o-nd fijn, dnd on the sparsity levels n^ and He- 



Proof. The proof is given in [Appendix C 



The recovery condition ^ provides a sufficient condition on the number 
Ux of dominant entries of x, for which BP-RES can stably recover x from z. 
The condition depends on the coherence parameters /i^, /if,, and /i^, and the 
number of sparse corruptions Ue- As for the case of DR, the situation that 
guarantees that the largest number n^ of dominant coefficients in x will be 
recovered stably using BP-RES, is when A and B are maximally incoherent 
ONBs. In this situation, the recovery condition (|9| reduces to 2nxne < M, 
which is two times more restrictive than that for DR (see [12] for a detailed 
discussion on this factor-of-two penalty). 

The following observations are immediate consequences of Theorem |3j 
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• If the vector x is perfectly n^^-sparse, i.e., ||x — X;t'||^ = 0, and no 
measurement noise is present, i.e., e = 0, then BP-RES using rj = 
perfectly recovers x whenever ^ is satisfied. Note that two different 
restoration procedures were developed for this particular setting in |12| 
Thms. 4 and 5]. Both methods enable perfect recovery under exactly 
the same conditions (cf. Table fl|. Hence, generalizing the recovery 
procedure to approximately sparse signals and measurement noise does 
not incur a penalty in terms of the recovery condition. 

• The restoration method in \n\ Thm. 5] requires a column-normalization 
procedure to guarantee perfect recovery under the condition (pi). Since 
in this special case, BP-RES (with 77 = 0) corresponds to BP, Theo- 
rem [3] implies that this normalization procedure is not necessary for 
guaranteeing perfect recovery under ([9]). Note, however, that this ob- 
servation does not apply to orthogonal matching pursuit (see [39j for 
more details). 

We finally point out that the recovery condition in ^ was shown in |12| 
to be tight for particularly-chosen comb signals in the case where A is the 
Fourier matrix and B = Im- 

3.3. BP separation: No knowledge on the support sets 

We finally consider the case where no knowledge about the support sets 
of the approximately sparse vectors x and e is available. A typical appli- 
cation scenario is signal separation jH E], e.g., the decomposition of audio, 
image, or video signals into two or more distinct features, i.e., in a part 
that exhibits an approximately sparse representation in the dictionary A 
and another part that exhibits an approximately sparse representation in B. 
Decomposition then amounts to performing simultaneous recovery of x and 
e from z = Ax + Be + n, followed by computation of the individual signal 
features according to Ax and Be. The main idea underlying this signal- 
separation approach is to rewrite (fTl) as 

z = Dw + n (10) 

where D = [ A B ] is the concatenated dictionary of A and B and the stacked 
vector w = [x"^ e^]-'". Signal separation now amounts to performing BPDN 



on ( 10 ) for recovery of w from z, which is also known as the synthesis separa- 



tion problem for microlocal analysis (see |10lllT] and the references therein). 
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A straightforward way to arrive at a corresponding deterministic recovery 
guarantee for this problem is to consider D as the new dictionary with the 
dictionary coherence defined as 

/irf = max Idfdjl = max {fia, fJ'b, fJ-m} ■ (H) 



One can now use BPDN to recover w from (10) and invoke Theorem [I] with 
the recovery condition in ([2]), resulting in 

w = n^. + ne< -(l + — ) . (12) 



However, it is important to realize that (12) ignores the structure underlying 
the dictionary D, i.e., it does not take into account the fact that D is a 
concatenation of two dictionaries that are characterized by the coherence 



parameters /la, fJ^b, and fim- Hence, the recovery guarantee (12) does not 
provide insight into which pairs of dictionaries A and B are most useful for 
signal separation. The following theorem takes into account the structure 
underlying D, enabling us to gain insight into which pairs of dictionaries A 
and B support signal separation. 

Theorem 4 (BP separation). Let z = Dw + n, with D = [ A B], w = 
[a-^ e^]"^, and ||n||2 < e. The dictionary D is characterized by the coherence 
parameters fia, ^^b, fJ-m, and fid, and we assume /if, < fia without loss of 
generality. Furthermore, let W = supp^{w). If 

J 2{1 +fia) l + /id . ,^„s 

w < max <^ ^^^^^ , — ^ , (13) 

fXa + 2/Xrf + ^fxl + /i^ 2fi, 

then the solution w of BP separation 




(BP-SEP) 



minimize w 



1 



subject to ||z — Dw||2 < f] 
using e < rj satisfies 

||w - w||2 < C7{e + r]) + Cs\\w - ww||^ , (14) 

with the (non-negative) constants C-j and Cg. 
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Proof. The proof is given in Appendix D 



The recovery condition (13) refines that in (12); in particular, considering 



the two-ONB setting for which /!„ = /i;, = and Hm = fJ^d- In this case, the 



whereas the one for BP separation (13) is 



straightforward recovery condition (11) corresponds to w < {1 + l/yUrf)/2 



«; < -^. (15) 



Hence, (13) guarantees the stable recovery for a larger number of dominant 
entries w in the stacked vector w. Recovery guarantees for perfectly sparse 
signals and noiseless measurements in the two-ONB setting were developed 
in |23 | l2 ^ H2]. The corresponding recovery condition w < {\/2 — 0.5)/ fXd turns 
out to be better (i.e., less restrictive) than the recovery condition for approxi- 



mately sparse signals and measurement noise provided in (15). Whether this 
behavior is a fundamental result of considering approximately sparse signals 
and noisy measurements or is an artifact of the proof technique remains an 
open research problem. 

4. Recovery Guarantees from the RIP-Framework and Comparison 

In this section, we develop alternative recovery guarantees for BP-RES 
and BP-SEP using results from the restricted isometry property (RIP) frame- 
work. We furthermore provide a comparison with the (coherence-based) re- 
covery guarantees obtained in the previous section and the guarantees for 
perfectly sparse signals and noiseless measurements presented in [T2[ [T8] (re- 
call Table [!}. 

4.I. RIP-based recovery guarantees 

An alternative way of obtaining deterministic recovery guarantees for ap- 
proximately sparse signals and measurement noise, i.e., for z = Ax + n, has 
been developed under the RIP framework |3^ HStilT] . There, the dictio- 
nary A is characterized by restricted isometry constants (RICs) instead of 
the coherence parameter fia- 

Definition 1 ([34j). For each integer n^ > 1, the RIC 6n^ of A is the 
smallest number such that 

(l-(5„J||x||^<||Ax||^<(l + (5„J||x||^ (16) 

holds for all perfectly n^-sparse vectors x. 

15 



The corresponding recovery conditions, which guarantee stable recovery 
of X with ||n||2 < e using BDPN, are, e.g., of the form i) 52n^ < V2 — 1 jB] or 
ii) 5n^ < 0.307 |47) . The main issue with such recovery conditions is the fact 
that the RICs cannot be computed efficiently, in general. In order to arrive 
at recovery conditions that are explicit in the number of nonzero entries n^, 



one may bound the RIC (16) from above using the coherence parameter /x^ 
as IHHIlS] 



Sn, < IJ'ainx - 1) 



(17) 



which can then be used to arrive at the corresponding recovery conditions 
i) Tix < {1 + ("\/2 — l)//ia)/2 or ii) n^ < 1 + 0.307 /na- Both of these recov- 
ery conditions are more restrictive than that in ([2]). Nevertheless, recovery 
guarantees obtained through the RIP framework turn out to be useful for 
deriving probabilistic recovery conditions (guaranteeing recovery with high 
probability) in the field of compressive sensing (CS) pSlfJQ]. The coherence- 



based bound in (17) is useful also because it puts limits on the RIC that are 
explicit in Ux- 

4-2. Recovery guarantees for sparsely corrupted signals 

We next provide coherence-based bounds on the RIC constants for BP 
restoration and BP separaion, and derive corresponding alternative recovery 
guarantees using results obtained in the RIP framework [44^ .47] . 

Recovery guarantee for BP restoration. As a byproduct of the proof for BP- 
RES detailed in Appendix C , the following coherence-based upper bound on 
the RIC for the matrix A = R^A was obtained: 



Lemma 5 (RIC bound for A). Let A = R^-A with R^- = Im 
For each integer Ux > I, the smallest number 6n^ such that 



BeBl 



{l-Sr, 



I l|2 ^ 



Ax 



<a + Sn 



holds for all perfectly rix-sparse vectors x G C^", is bounded from above by 



Sn^ < fJ'airix - 1) + 



nxrieK 



[1 - Ubijie - iy\ 



(18) 
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Combining the bound (18) with the condition 5n^ < 0.307 from [17] en- 



ables us to obtain the following recovery condition: 

^^ ^ (0.307 + /xJ[l-/XfcK-l)]+ ^^g^ 

that guarantees the stable restoration of x from R^-z = Ax + R^-n using BP- 



RES. We emphasize that for //„ < 1 - 2 ■ 0.307 ^ 0.386 the condition ([19j) is 
more restrictive than the condition ([9]), which reads 

1 (l + /i„)[l-^,(ne-l)] + 



2/x„[l-/i6(ne- 1)] +ne/i^ 



Hence, for most relevant values of the coherence parameter /i^, the recovery 
condition in ^ is superior (in a sense of enabling the recovery of a larger 
number n^ of dominant coefficients in x) to that obtained using the RIP 
framework. 

Recovery guarantee for BP separation. During the proof for BP separation 



detailed in [Appendix D[ the following coherence-based upper bound on the 



RIC for the concatenated dictionary D = [ A B ] was obtained. 

Lemma 6 (RIC bound for D). Let D = [A B] he characterized by fia, 
lily, firn, o-nd fid, and assume fii, < fia without loss of generality. For each 
integer w > 1 the smallest number 6^^ such that 

(1 - (5^) ||w||2 < ||Dw||2 < (1 + (^«,) ||w||2 

holds for all perfectly w-sparse vectors w G C^"^^**, is bounded from above 
by 

6^ < min I - (^/i„(w; - 2) + w v^/^Tp^ j , /id(w - 1) [ • (20) 



As in Section 4.2, we use the right hand side of (20) in combination with 
the recovery condition 6^ < 0.307 |S7J, to obtain a recovery guarantee for 
signal separation. In particular, BP-SEP guarantees stable recovery if 

f 2(0.307 + /ia) , 0.307 , 
u;<max ^-=^,1 + -— ^. (21) 
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Figure 1: Comparison of the recovery conditions for fj,„i = A*d = 0.1 and fia = ^J-b = 0.04. 
As in the case of BP restoration, this condition turns out to be more re- 



strictive than that in (13) for most relevant cases. However, the difference 



between the recovery condition in (21) and that in (13) is small, in general 

and fim 



To see this, consider the two-ONB case, i.e., fia 



l^b 



l^d- 



Here, the recovery condition (21 ) corresponds to w < 0.614/yU(^, which is only 



slightly more restrictive than the condition w < 2/(3/id) given in (15). 



4-3. Comparison of the recovery guarantees 

Fig. [I] compares the recovery conditions for the general model ([I]) to those 
obtained in [T2| [T8] for perfectly sparse signals and noiseless measurements 
(see also Table fl|. We set /i^ = /i^ = 0.1 and Ha = f^b = 0.04, and compare 
the three different cases analyzed in Section [3j The following observations 
can be made: 

• Direct restoration: For DR, no recovery guarantees are available through 
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the RIP framework. The recovery conditions for the general model (fTl) 
detailed in (JTl) and that in [12, Eq. 11] for perfectly sparse signals 
and noiseless measurements coincide. Hence, the generalization to ap- 
proximately sparse signals and measurement noise does not entail a 
degradation in terms of the corresponding recovery condition. 

• BP restoration: In this case, the recovery conditions for the general 
setup considered in this paper and the condition [T2| Eq. 14] for per- 
fectly sparse signals and noiseless measurements also coincide. Hence, 
generalizing the results does not incur a loss in terms of the recov- 



ery conditions. As mentioned in Section 4.2, the recovery condition 



obtained trough the RIP framework turns out to be more restrictive. 

BP separation: In this case, we see that all of the recovery conditions 
differ. In particular, the condition [TBI Eq. 13] for perfectly sparse 
signals and noiseless measurements is less restrictive that that for the 
general case ([I]). Furthermore, the recovery condition (21) obtained in 



the RIP framework turns out to be the most restrictive. However, the 
differences between the three recovery conditions are rather small. 

In summary, we see that having more knowledge on the support sets prior 
to recovery yields less restrictive recovery conditions. This intuitive behavior 
can also be observed in practice and is illustrated in Section |5j 

We finally emphasize that all of the recovery conditions derived above are 
deterministic in nature and therefore conservative in the sense that, in prac- 
tice, recovery often succeeds for sparsity levels n^ and Ug much higher than 
the corresponding guarantees indicate. In particular, it is well-known that 
deterministic recovery guarantees are fundamentally limited by the so-called 
square-root bottleneck, e.g., [Ill [18], [5^, as they are valid for all dictionary 
pairs A and B with given coherence parameters, and all signal and interfer- 
ence realizations with given sparsity levels n^ and n^. Nevertheless, we next 
show that our recovery conditions enable us to gain considerable insights 
into practical applications; i.e., they are useful for identification of appropri- 
ate dictionary pairs (e.g., being sufficiently incoherent) that should be used 
for signal restoration or separation. 

5. Application Examples 

We now develop two application examples to illustrate the main results 
of the paper. First, we show that direct restoration, BP restoration, and BP 
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separation can be used for simultaneous denoising and declicking of corrupted 
speech signals. Then, we illustrate the impact of support-set knowledge for 
a sparsity-based in-painting application. 

5.1. Simultaneous denoising and declicking 

In this example, we attempt the recovery of a speech signal that has been 
corrupted by a combination of additive Gaussian noise and impulse noise. 
To this end, we corrupt a 9.5 s segment (44100 kHz and 16 bit) from the 
speech signal in |51] by adding zero-mean i.i.d. Gaussian noise and impulse 
noise. The variance of the additive noise is chosen such that the mean- 
squared error (MSE) between the L-dimensional original audio signal y and 
the noisy version y, defined as 

MSE = 101ogio(||y-y||2/^), 

is —30 decibel (dB). The impulse interference (used to model the clicking 
artifacts in the audio signal) is generated as follows: We corrupt 10% of the 
samples and chose the locations of the random clicks, which are modeled by 
the interference vector e, uniformly at random. We then generate the clicks 
at these locations by adding i.i.d. zero-mean Gaussian random samples with 
variance 0.1 to the noisy signal (whose maximum amplitude was normalized 
to +1). 

Recovery procedure. Recovery is performed with overlapping blocks of di- 
mension M = 1024. The amount of overlap between adjacent blocks is 64 
samples. We set A to the 1024 x 1024-dimensional DCT matrix, B = Im, 
and perform recovery based on z = Ax -f- e -f n. The main reasons for using 
the DCT matrix in this example are i) the speech signal is approximately 
sparse in the DCT basis and ii) the mutual coherence between the identity 
and the DCT is small, i.e., /i^ = l/v5l2, which leads to less restrictive 
recovery conditions ([T]), (tol), and (13). For all three recovery methods, we 



first compute an estimate x of x (and of e in the case of BP separation) 
and then compute an estimate of the denoised speech signal according to 
y = Ax. In order to reduce undesired artifacts occurring at the boundaries 
between two adjacent blocks, we add and overlap the recovered blocks using 
a raised-cosine window function when resynthesizing the entire speech signal. 
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Figure 2: Mean squared error (MSE) results of simultaneous reduction of Gaussian and 
impulse interference in a corrupted speech signal (the x-axes correspond to sample indices, 
the y-axes to magnitudes). 
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Discussion of the results. Fig. [2] shows snapshots of the corruption and recov- 
ery procedure and the associated MSE results. The corrupted signal, which 
is impaired by Gaussian noise and impulse noise, has an MSE of — 19.7dB. 
The individual results of the three recovery procedures analyzed in this paper 
are as follows: 



Direct restoration: In this case (see Fig. 2(d)), the locations £ of the 
impulse noise realizations are assumed to be known prior to recovery. 
We set rY = {1, . . . , 192}, assuming that the speech signal occupies only 
the lowest 192 frequencies. The MSE between the original signal and 
the one recovered through DR corresponds to — 34.7dB and, hence, DR 
is able to improve the MSE by 15 dB (compared to the signal that is 
impaired by Gaussian and impulse noise). 

BP restoration: Here, we assume that the locations of the impulse 
noise spikes S are known prior to recovery but nothing is known about 
X (except for the fact that it allows for an approximately sparse rep- 
resentation in A). We perform BP restoration with t] = 0.6, which 



results in an MSE of — 33.3dB (see Fig. 2(e)). Note that the parame- 
ter 7] determines the amount of denoising (for the Gaussian noise) and 
can be used to tune the resulting MSE. 

• BP separation: In this case we assume that nothing is known about 
the support sets of either x or e. We use BP separation with rj = 0.4 
and discard the recovered error component e; the resulting MSE corre- 



sponds to —31.6 dB (see Fig. 2(f) ). As it is the case for BP restoration, 
T) determines the amount of denoising (of the Gaussian noise compo- 
nent). BP separation achieves surprisingly good recovery performance 
(compared to DR and BP-RES), while being completely blind to the 
locations of the sparse corruptions. Hence, BP separation offers an 
elegant way to mitigate impulse noise in speech signals, without re- 
quiring sophisticated algorithms that detect the locations of the sparse 
corruptions. 

This application example shows that more knowledge on the support sets 
X and/or £ leads to improved recovery results (i.e., smaller MSE). This 
intuitive behavior has been observed in [12] for perfectly sparse signals and 
noiseless measurements. We emphasize that DR, BP restoration, and BP 
separation are all able to simulatenously reduce Gaussian noise and impulse 
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interference as the resulting MSEs are all smaller than —30 dB (corresponding 
to the MSE of the noisy signal). The recovery procedure one should use in 
practice depends on the amount of support-set knowledge available prior to 
recovery. 

We finally note that reduction of noise and clicks in impaired audio sig- 
nals is a well-studied topic in the literature (see, e.g., [2| and references 
therein). However, most of the established methods rely on Bayesian es- 
timation techniques, e.g., [21 [52l [53], which differ from the sparsity-based 
approach proposed here. Sparsity-based algorithms for restoration of im- 
paired audio signals have been proposed recently in [T^ [T7] : however, no 
recovery guarantees are available for the proposed OMP-based restoration 
algorithm. We emphasize that virtually all proposed methods require knowl- 
edge of the locations of the sparse corruptions prior to recovery, whereas our 
results for BP separation show that sparse errors can effectively be removed 
blindly from speech signals. The main goal of this example is to illustrate 
the performance our algorithms and not to benchmark the performance rela- 
tive to existing methods; a detailed performance and restoration-complexity 
comparison with existing methods for simultaneous denosing and declicking 
is left for future work. 

5.2. Removal of scratches in old photographs 

We now consider a simple sparsity-based in-painting application. While 
a plethora of in-painting methods have been proposed in the literature (see, 
e.g., |6|[8]-[T0] and the references therein), our goal here is to not to benchmark 
our performance vs. theirs but rather to illucidate the differences between 
BP restoration and BP separation, i.e., to quantify the impact of support- 
set knowledge on the recovery performance. In this particular example, we 
intend to remove scratches from old photographs. To this end, we corrupt a 
512 X 512 greyscale image by adding a mask containing artificially generated 
scratch patterns. This procedure corrupts 15% of the image. In order to 
demonstrate the recovery performance for approximately sparse signals, the 
image was not sparsified prior to adding the corruptions (which is in contrast 
to the in-painting example shown in [12j). The MSE between the original 



photo (shown in Fig. 3(a)) and the corrupted version (shown in Fig. 3(b)) 



corresponds to — 16.7dB. 

Restoration procedure. Scratch removal proceeds as follows. For BP restora- 
tion and BP separation, we assume that the image admits an approximately 
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(a) Original (courtesy of NASA jS^) (b) Corrupted photo (MSE = -16.7dB) 




(c) Result of BP-RES (MSE = -29.2 dB) (d) Result of BP-SEP (MSE = -19.4dB) 

Figure 3: Example of using BP restoration (BP-RES) and BP separation (BP-SEP) for 
removal of scratches from old photographs. 
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sparse representation in the two-dimensional DCT basis A, whereas the in- 
terference is assumed to be sparse in the identity basis B = Im- Recovery 
is performed on the basis of the full 512 x 512 pixel image, i.e., we have 
M = 512^ corrupted measurements. For BP restoration, we assume that 
the locations of the scratches are known prior to recovery, whereas no such 
knowledge is required for BP separation. For BP restoration we recover x 
(for BP separation we additionally recover e) and then compute an estimate 
of the image as y = Ax. Since we consider noiseless measurements, we 
set T) = for both recovery procedures. Note that DR is not considered in 
this example as information on the location of the dominant entries of x is 
difficult to acquire in practice. 

Discussion of the results. Fig. [3] shows results of the corruption and recov- 
ery procedure along with the associated MSE values. For BP restoration, 
we see that the recovered image has an MSE of — 29.2dB and well approx- 
imates the ground truth. For BP separation, the MSE improves over the 
corrupted image, but in parts where large areas of the image are corrupted, 
blind removal of scratches fails. Hence, knowing the locations of the sparse 
corruptions leads to a significant advantage in terms of MSE and is therefore 
highly desirable for sparsity-based in-painting methods. 

We finally note that the recovery conditions (JTl), (|9]), and (13) turn out to 



be useful in practice as they show that the dictionary A must both i) sparsify 
the signal to be recovered and ii) be incoherent with the interference dictio- 
nary B. Note that the second requirement is satisfied for the DCT-Identity 
pair used here, whereas other transform bases typically used to sparsify im- 
ages (i.e., to satisfy the first requirement), such as wavelet bases, exhibit 
high mutual coherence with the identity basis. Hence, our recovery guaran- 
tees help to identify good dictionary pairs for a variety of restoration and 
separation problems. 

6. Conclusions 

In this paper, we have generalized the results presented in [T2l [T8] for 
the recovery of perfectly sparse signals that are corrupted by perfectly sparse 
interference to the much more practical case of approximately sparse sig- 
nals and noisy measurements. We proposed novel restoration and separation 
algorithms for three different cases of knowledge on the location of the dom- 
inant entries (in terms of absolute value) in the vector x, namely 1) direct 
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restoration, 2) BP restoration, and 3) BP separation. Moreover, we devel- 
oped deterministic recovery guarantees for all three cases. The application 
examples demonstrated that our recovery guarantees explain which dictio- 
nary pairs A and B are most suited for sparsity-based signal restoration or 
separation. Our comparison of the presented deterministic guarantees with 
similar ones obtained using the restricted isometry property (RIP) frame- 
work and to those provided in [T2| [T8] reveals that, for BP restoration and 
BP separation, considering the general model does not result in more restric- 
tive recovery conditions. For BP separation, however, the recovery conditions 
for the general model considered here turn out to be slightly more restrictive 
as it is for perfectly sparse signals and noiseless measurements. 

There are many avenues for follow-on work. The derivation of proba- 
bilistic recovery guarantees (with randomness in the signals rather than in 
the dictionaries) leading to recovery conditions guaranteeing restoration and 
separation with high probability is an interesting open research topic. Fur- 
thermore, a detailed exploration of more real-world applications using the 
restoration and separation techniques analyzed in this paper is left for future 
work. 

Appendix A. Proof of Theorem [l] 

The proof follows closely that given in [31| Thm. 2.1] and relies on tech- 
niques developed earlier in [201 1311 113] . 

Appendix A. 1. Prerequisites 

We start with the following definitions. Let h = x — x, where x denotes 
the solution of BPDN and x is the vector to be recovered. Furthermore, 
define ho = P^rh with the set X = supp„Jx). The proof relies on the 
following facts. 

Cone constraint. Let cq = 2||x — :s^x\\i with :six = Pa'X; then pOf HSj H6] 

||h-ho||i < ||ho||i + eo (A.l) 

which follows from the fact that BPDN delivers a feasible solution x satisfying 
||x||;^ > ||x||j^ and from 

||x||i > ||x||^ = IIxa-IIi + ||xa'<:||i = IIxa" + holli + ||h - ho + h;fc||^ 

> ||x;t|li - llholli + ||h - holli - ||x;t"=||^ . 
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Application of the reverse triangle inequality to the left-hand side term 

(A.2) 



of (A.l) yields the following useful bound: 

||h||i < 2||ho||i + eo 



Tube constraint. We furthermore have H3l HE 



lAhl 



|Ax-y- (Ax-y) 



< ||Ax-y| 



I Ax — y||2 < r] + e. 



(A.3) 



Coherence-based restricted isometry property (RIP). Since ho is perfectly n^- 
sparse, Gersgorin's disc theorem [55^ Thm. 6.1.1] applied to ||Aho||2 yields 



;i - /U,K - l))l|ho|l2 < IIAholl' < (1 + f-ia{n., - l))||h, 



0II2 



(A.4) 



Appendix A.2. Bounding the error ||ho||2 on the signal support 

The goal of the following steps is to bound the recovery error ||ho||2 on 
the support set X. We follow the steps in [31] to arrive at the following chain 
of inequalities: 



,H xH 



H kH 



h^A^Aho > hS'A^Aho - (h-ho)''A^Ah, 



> (1 - Hain^ - 1)) llholls - 



J2T.^K]k^kMK 



keX i£X<^ 



(A.5) 



> (1 - Hain.^ - 1)) ||ho||2 - /Ua||ho||i ||h - holli (A.6) 

> (1 - fia{n^ - 1)) ||ho||2 - yUa||ho||i (||ho||i + eo) (A. 7) 

> (1 - fia{nx - 1)) ||ho||2 - /ia^x||ho||2 - /iaA/^||ho||2 Cq 

(A.8) 
= (1 - fiai^n^ - 1)) llholla - /ia v^l|ho|l2 eo, (A.9) 



where (A.5) follows from (A.4), (A.6) is a consequence of a^a^ < /i^, VA; 7^ 



(A. 7) results from the cone constraint (A.l), and (A.8) from the Cauchy- 



Schwarz inequality. We emphasize that (A.9) is crucial, since it determines 



the recovery condition for BPDN. In particular, if the first RHS term in (A.9 ) 



satisfies (1— /ia(2n2^ — 1)) > and ho 7^ Oat^xi, then the error ||ho||2 is bounded 
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from above as follows: 

Ih^A^Ahol +;U„^/ri;;||ho||2eo 



|ho|l2 < 



< 



< 



(1 ~ fiaC^n^- 1)) llholls 
II Ahll^ II Aholl^ + fia./rT;;:\\ho\\^ eg 

(l-/ia(2n,-l))||ho||2 

{e + r])^/lT~ijJJh^^^\\ho\\^ + /iaA/^||ho||2eo 



(1 -/ia(2n^- 1)) ||ho||2 
1 - Ha{2n^ - 1) 



(A.IO) 
(All) 
(A. 12) 
(A.13) 



Here, (A.IO) is a consequence of (A.9), (A. 11) follows from the Caucliy- 



Schwarz inequality, and (A. 12) results from the tube constraint (A. 3) and 



the RIP (A. 4). The case ho = Oat^xi is trivial as it implies ||ho||2 = 0. 



Appendix A. 3. Bounding the recovery error ||h||2 

We are now ready to derive an upper bound on the recovery error ||h||2. 
To this end, we first bound ||Ah||2 from below as in |3l] 



|Ah||^ = h^A^Ah = 5^[h^],af a,[h]. 



ki 



J2\Mlmkf+ 5^ [h^],afa,[h] 

k k/,kjte 



>llh| 



fx 



. E llh"! 



(A, 14) 



k,i,k^i 
|h||^+/Xa$^|[hUr-/..5^|[h^].[h]. 



'l + /ia)||h||2 -/ia||h||i 



kl 



where (A. 14) follows from ||afc||2 = 1, V/c, and |a^a£| < fia, VA; ^ 



(A.15) 
, With 



(A.15), the recovery error can be bounded as 



IIAIII^ I 111 11-^ 

2 / Ah 2 + /i„ h^ 
|h||2 < z 



/ia 



< 



(£ + r/)^ + /ia(2||ho||, + eo)^ 

1 +/ia 



(A.16) 
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where (A. 2) is used to arrive at (A. 16). By taking the square root of 



(A. 16) and applying the Cauchy-Schwarz inequality, we arrive at the fol- 



lowing bound: 



|h||2< 



< 



[e + r])^ + fia{2\\ho\\^ + eo] 






(A.17) 



Finally, using ||ho||;^ < y^||ho||2 with the bound in (A. 13) followed by alge- 
braic simplifications yields 



|h|l2< 



{e + T]) + ^a (2yn;;||ho||2 + eo) 



< (e + r/) 
+ eo 



1 - /ia(2n^ - 1) + I^JjjT^^l + ^a{n^ - 1) 

v/rT7^(i-;u,(2n,-i)) 



(l-/i„(2n,-l)) 
Co(^ + £^) + Ci||x-xa'||i 



which concludes the proof. We note that by imposing a more restrictive 
condition than n^; < (1 + l/yUa)/2 in (|2|, one may arrive at smaller constants 
Co and C\ (see [IB] for the details). 

Appendix B. Proof of Theorem [2] 

The proof is accomplished by deriving an upper bound on the resid- 
ual errors resulting from direct restoration. Furthermore, we show that the 
recovery condition (7| guarantees the existence of R^- = \m — B^B^ and 

(R£A^)t. 

Appendix B.l. Bounding the recovery error 

We start by bounding the recovery error of DR as 



||X-X||2 < ||x;t -Xa-IIs 



X;t'c — X;t'c||2 



< \\yix -xa-IIs + IIxa'^i- 



(B.l) 
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The only term in (B.l) that needs further investigation is \\'^x — ^x\\2- ^s 
shown in (Is]), we have 

[±]x = (Rf A;t)^Rf z = X;t + (R^Ax)^R^(Ax;,c + n) 

and hence, it follows that 

||XA'-X;t||2< ||(R^A;f)^Rfv||2, (B.2) 

where v = Ax^^^c + n represents the residual error term. The remainder of 



the proof amounts to deriving an upper bound on the RHS in (B.2). We 
start with the definition of the pseudo-inverse 



R^A;t)^R^v| 



< 



(A^R^A;,)-iA^R^v||2 

(A^RfA;,)-HU|AfR,v|L 



(B.3) 
(B.4) 



where (B.3) is a consequence of R^- = R^^R^:, and (B.4) follows from the 
Rayleigh-Ritz theorem [551 Thni. 4.2.2]. We next individually bound the 



RHS terms in (B.4) from above 



Appendix B.2. Bounding the i2-norm of the inverse 



The bound on the norm of the inverse in (B.4) is based upon an idea 
developed in [2l]. Specifically, we use the Neumann series (I|;f| — K)~^ = 
^W + E^iK^ ISSl Lem. 2.3.3] to obtain 



(A^R^A;,) 



[AfA^t-A^B^BtA;^)-' 



H^l 



K) 



fc=i 



<1 + 



E^^ 



fc=i 



iw + E^' 

k= 

oo 

<H-El|K| 



fc=l 



IKI 



which is guaranteed to exist whenever ||K|L < 1 with 



(B.5) 



K 



i\x\ 



H 



H-, 



- A'^Kx + A5B^B^A;t. 



30 



We next derive a sufficient condition for which ||K||2 < 1 and, hence, the 



matrix A^R£-A;f is invertible. We bound 11 K 11 2 from above as 



|K||2<||I|A^|-A^A;,||2 



AfB^BjA;, 



< 



IJ'a[nx 



l) + ||AfB£(BfB£)-iBfA;,| 



2 ' 



(B.6) 
(B.7) 



where (B.6) results from the triangle inequality and (B.7) is a consequence 



of Gersgorin's disc theorem [55l Thm. 6.1.1] applied to the £2-iiorm of the 
hollow matrix I\;^\ — A^A;^'. We next bound the RHS term in (B.7) as 

-iB|^A;,||^ < ||A^B,||^ ||(B|^B,)-i||^ ||Bf A;,||^ (B.8) 

'% (B.9) 



A;t'B£-(B^ Bf, 



12 
<r II A^R 11^ 



B?B 



< n^rieiJ,^^ 



(BfB, 



11 



< 



^x^efJ'r, 



Amin(B£- B 



< 



^x^efJ'r, 



e^Sj 



[1 - fib{ne- 1)] 



+ ' 



(B.IO) 
(B.ll) 



where (B.8) follows from the £2-HicLtrix-norm bound, (B.9) from A;^^^ 



A^B^TTand llA^B^ 



IF 



< 



B^Ax\\p, and ( |B.10| ) from 



iA?B.ii;=5:$:Kb,r<$:^^: 



'^x'^et^r, 



kex ie£ 



kex iee 



Note that (B.ll ) requires rif, < 1 + l//ib, which provides a sufficient condition 
for which the pseudo-inverse B^ exists. 

Combining (B.5), (B.7), and (B.ll) yields the upper bound 



||(AfR^A;t) 

which requires 



< 



1 - fJ'ainx - 1) 



nxne^lfn 



[1-M6(ne-1)]" 



/ia(^i 



'^x'^eh^ri 



>0 



(B.12) 



(B.13) 



[1 - fibirie - l)i 

for the matrix (A^RfA;^')^^ to exist. We emphasize that the condition 
(B.13) determines the recovery condition for DR ([T]). In particular, if 

(1 - HaiUx - 1)) [1 - flb{ne - 1)]^ > nxUeHl, 



then (B.13) and n^ < 1 + l/yUfc are both satisfied and, hence, the recovery 



matrix (RsAxYti-s required for DR exists. 
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Appendix B.3. Bounding the residual error term 

We now derive an upper bound on the RHS residual error term in (B.4) 
according to 



H 



H, 



A^R£v||2<||A^v||2 + 



< V?T'a;||v||2 






where (B.14) is a result of 



lA^vll 



'El^N'< JEll^'^ll2lMl2<x/^IM 



2 • 



(B.14) 



(B.15) 



kex 



kex 



The bound on the second RHS term in (B.14) is obtained by carrying out 



similar steps used to arrive at (|B.11|), i.e. 

l^B.bJ. 



< A^B,U(B^B^ 



^^ ^-^l| llB^vll 

1 2 W £ II 2 



< 



< 



^/n~ri4i: 



m WrtH 



F V 



A^i„(BfB£)ll"^ Il2 



n, 



;\AW^ 



12 • 



[l-/i5(ne-ir 

Finally, we bound the £2-norm of the residual error term according to 



:b.i6) 



Axvc + nlL < IIAxvc 



■A"^ll2 



|n||2 < IIxa-cIIi + ||n| 



(B.17) 



smce 



I Ax 



^""=112 



y^ ^k[y^]k 



keX'^ 



< ^ ||aA:[x]fe||2 = IIxa-cIIi 
2 fe6A'= 



Appendix B.4- Putting the pieces together 

In order to bound the recovery error on the support set X, we combine 



(|R12j) with ( |B.14[ ) and ( |B.16[ ) to arrive at 

([1 - Hb{ne - 1)]^ + ne/irn) 



(R£A;t)^R^v|l < 



{1 - fiain., - 1)) [1 - ^b{ne - 1)] -n^rieHl 
cllvIL . 



(B.18) 
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Finally, using (B.17) in combination with (B.l), (B.2), and (B.18) leads to 



|x — xlL < ce + (c + 1] 



xa-IIi 



C3e + Ci\\^-:six\\i 



which concludes the proof. 



Appendix C. Proof of Theorem [3] 

We first derive a set of key properties of the matrix A = R^A, which are 



then used to proove the main result following the steps in [Appendix A 



Appendix C.l. Properties of the matrix A 

It is important to realize that BP restoration operates on the input-output 
relation 



R^-z = R£- (Ax + Be£- + n) = Ax + Hgn 



(C.l) 



with R^: = \m — B^-B^ and A = R^-A. The recovery condition for BP 
restoration ([9]), which will be derived next, also ensures that R^- exists; this 
is due to fact that the recovery condition for DR ([T]) ensures that R^- exists 
and the condition for BP restoration (|9| is met whenever ([T]) is satisfied. 

In order to a dapt the proof in Appendix A for the projected input-output 
relation in (C.l), the following properties of A are required. 



Tube constraint. Analogously to Appendix A we obtain 



Ah 



< ||R£(Ax 



R^(Ax 



< r/ + ||R£-n||2 < r/ + e. 



where the last inequality follows from the fact that R^- is a projection matrix 
and, hence, 



iRpnL < llnlL < e. 

I (^ W 2 II Hz 



Coherence-based bound on the RIC. We next compute a coherence-based 
bound on the RIC for the matrix A. To this end, let ho be perfectly n^- 
sparse and 



Ahr 



h^A^Aho - h^A^B^BjAho 



<(l + /i,(n, -l))||ho||2 + 



h^A^B^BjAho 



(C.2) 
(C.3) 



33 



where (C.2) follows from R^^R^- = R^- and (C.3) from Gersgorin's disc the 



orem |55l Thm. 6.1.1]. Next, we bound the second RHS term in (C.3) as 
follows: 



h^A^B^B^Aho 



= |h^A^B£(B^B£)-^B^Aho| 

<A-L(BfB,)||B?Ahn 
,(BfB, 



|2 
|2 



<A-L(B?B£)||BfA;,||2||ho||2 



< 



'^x'^eH'ri 



[1 - fibirie - 1)] 



111 II z 
+ II^0||2 



(C.4) 
(C.5) 

(C.6) 



where (C.4) follows from |55l Thm. 4.2.2], (C.5) from the i'2-norm inequality. 



The inequality (C.6) results from 

||B£ A;t||2 < |P£- A;t||^ 



^m- 



iee kex 



Note that (C.6) requires rig < 1 + l//if,, which is a sufficient condition for 



(B^'^B^-) ^ to exist. Note that n^ < 1 + ^/fJ'b holds whenever the recovery 
condition for BP-RES in ([T]) is satisfied. Combining (C.3) with (C.6) results 



m 



||R£Aho||2<( l + /iaK-l) + - 
=(l+'^)l|ho||^ 

We next compute the lower bound as 

2 



'^x'^eH'rt 



Hb[ne 



1)]^ 



A0II2 



(C.7) 



Aho 



ho^A^Aho 



h^A^B^BjAho 



^0 



> (1 - fia{n^ - 1] 

> (1 - fia{n^ - I] 



10II2 



10II2 



h^A^B^BjAho 



^x'^e^n 



> 1 - IJ,a{na 



D- 



[1 - /ife(ne 

2 
^x^efJ'm 

[1 - iib{ne- l)]" 



llhol 



10II2 



(C.8) 
(C.9) 

(CIO) 



10II2 



where (C.8) follows from [551 Thm. 6.1.1] and (C.9) is obtained by carrying 



out similar steps used to arrive at (C.6). Note that (C.7) and (C.IO) provide 



a coherence-based upper bound 6 on the RIC of the projected matrix A 
R^A. 
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Upper bound on the inner products. The proof detailed in Appendix A re- 
quires an upper bound on the inner products of columns of the matrix A. 
For i y^ j, we obtain 



|afaj| = |afR^aj| < |afaj| + afB^B^aj 
</i,+ |afB£(BfB£)-iBfa,| 



< /ia + 



< /ia + 



|a,^B.B?a, 



[1 - fibiUe - 1)] + 
||"R^a.|| llR-f^pi.ll 

[i-/xfeK-i)]+ ' 



(C.ll) 
(C.12) 

(C.13) 



where (C.ll) follows from the definition of the coherence parameter /i^, 



(C.12) is a consequence of Gersgorin's disc theorem, and (C.13) from the 



Cauchy-Schwarz inequality. Since 






'5:ib?«.i' =^ •/ 



neK 



ke£ 



for all i 



, Na, the inner products with i ^ j satisfy 



^i ^j "Si f^a "T 



UeK 



[1 - I2h{ne-l)]' 



A 

= a. 



(C.14) 



Lower bound on the column norm. The last prerequisite for the proof is a 
lower bound on the column- norms of A. Application of the reverse triangle 
inequality, using the fact that ||aj||2 = 1, Vi, and carrying out the similar 



steps used to arrive at (|C.14) results in 



\~ l|2 
I «ii 1 1 2 



t\,F 3-i o ^ 3-,' 3-j 



> 1 



neK 



[1 - Ubirie -1)1 



afB.Bta, 



b. 



Appendix C.2. Recovery guarantee 

We now derive the recovery condition and bound the corresponding error 



The proof follows that of Appendix A For the sake of simplicity of 



exposition, we make use of the previously defined quantities (5, a, and b. 
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Bounding the error on the signal support. We start by bounding the error 



^0ll2 



as follows: 



h^A^Ahr 



> 



h;?A^Ahr 



H7H' 



XI 



10II2 



[h-ho)"A^Aho 



- an^,||ho||2 - aJn^\\ho\\^eo 



(C.15) 



cllhoL - av/ri;^||ho||2eo 



with 



C^ 1- 6 - aUx = 1 - fJ^aC^Ur, - 1] 



'^n^riei^: 



[I ~ fibirie - W 

Note that the parameter c is crucial, since it determines the recovery condi- 
tion for BP-RES (|9|. In particular, c > is equivalent to ^ 

[1 - /ia(2raa; - 1)]^ [1 - flb{ne - 1)]^ > 2na;ne/i^. 

If this condition is satisfied, then we can bound HhoUg from above as follows: 



Iholla < 



(e + 77) V 1 + (5 + ay/n^eo 



c 



Bounding the recovery error. We next compute an upper bound on ||h||2. To 

2 



this end, we start with a lower bound on 

2 



Ah 



as 



Ah 



> (6 + a)||h||2 - a||h||^ = (1 + /ia)||h||2 - a||h||^ 



since b + a = 1 + fia- Finally, we bound ||h||2 as follows: 



II,,, /. , ,c + 2,/an^Vl + 5 y/ay/T+Jt^ 

= 05(1] + e) + Cg||x-XA'||i, 

where the constants C5 and Cq depend on /Iq, fib, Ux, and Ue, which concludes 
the proof. 

Appendix D. Proof of Theorem |4] 

In order to prove the recovery guarantee in Theorem |4| we start by 
deriving a coherence-based bound on the RIC of the concatenated matrix 
D = [A B] which is then used to prove the main result. 
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Appendix D.l. Coherence-based RIC for D 



AB 



In this section, we obtain an equivalent bound to that in Appendix A.l 
for the dictionary D that depends only on the coherence parameters /i^, fij,, 
fim, and lid, and the total number of nonzero entries denoted hj w = rix + ne- 



Bounds that are explicit in n^ and rig. Let ho = [h^ h^ ] where h^. = 
P;f (x — x) and he = Psie — e) are perfectly n^ and n^. sparse, respectively. 
We start by the lower bound on the squared ^2-norm according to 



|Dho||^=[hf hf ] 



.H 



^Na 







Ia^. 



A^A A^B 
B^A B^B 

ho + h^ 



K 

he 

A«A-I^„ 
B^A 



A^B 
B^B-I 



Nt 



>llh, 



0II2 






A^B^ 



B>.A,c: B^^ B^- 



L|f| 



I0II2 ) 



ho 

(D.l) 



where (D.l ) follows from the reverse triangle inequality and elementary prop- 



erties of the £2 matrix norm. We next compute an upper bound on the matrix 



norm in (D.l) as follows 



A^A 







y\x\ 



< max{||A^A 





I 



B^B 



H^l J 



+ 



X 



\x\\ 





x^s 

f ^f -I 



A^B 



B^A, 



x^e 




IB^B 



1^1 1 



J 



Ia^r II 



(D.2) 



where (D.2) is a result of the triangle inequality for matrix norms and the 



facts that the spectral norm of both a block-diagonal matrix and an anti- 
block-diagonal matrix is given by the largest among the spectral norms of 
the individual nonzero blocks. The application of Gersgorin's disc theorem 



to the max{-}-term in (D.2) and 



\A^Be\\^<\\A^Be\\p< 



Z^Z^Iafb^ 



< \/rh: 



^ef^m 



kex iee 



leads to 



max{||Af Aa- -'i-\x\ 



\^£^£ 



H£\\ 



J 



I ^^Ti. II 



< max {nairix - 1), /^fe(rie - 1)} + Vn~n^ 
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Hence, we arrive at the following lower bound 

||Dho||2 > ||ho||5(^l -max{fia{n^- l),fib{ne- 1)} 



rixrielJ-f, 



(D.3) 



By performing similar steps used to arrive at (D.3) we obtain the upper 
bound 



iDh 



0II2 < \\ho\\l\^l + max {fiain^ - l),fib{ne- 1)} + ^/n~n^j . (D.4) 



Bounds depending on w = n^ + Ug. Both bounds in (D.3) and (D.4) are 



explicit in Ux and n^. Since the individual sparsity levels n^ and rig are 
unknown prior to recovery, a coherence-based RIP bound, which depends 
solely on the total number w = n^ + n^ oi nonzero entries of ho rather than 
on Hx and n^, is required. To this end, we define the function 



g{rix,ne) = max{ij,ainx - l),/i6(ne - 1)} + A/ra^ne/i^ 
and find the maximum 

g{w) = max g{nx,w -Ux). (D.5) 

0<nx<:W 

Since g{w) only depends onw = Ux + Ue and g{nx, Ug) < g{w), we can replace 



g{nx,ne) by g{w) in both bounds (D.3) and (D.4). 

We start by computing the maximum in (D.5). Assume fia{nx — 1) > 
fJ'b{ne — 1) and consider the function 



9: 



'a{nx, W-Ux) = flairix - 1) + \/l^x{w - Ux) fl^,. 



(D.6) 



It can easily be shown that ga{nx, w — Ux) is strictly concave in Ux for all < 
Ux < w and < w < oo and, therefore, the maximum is either achieved at 
a stationary point or a boundary point. Standard arithmetic manipulations 



show that the (global) maximum of the function in (D.6) corresponds to 

9aiw) = - i^fiaiw - 2) + W^/fil + /i^j . (D.7) 

For the case where fiai^x — 1) < /^fe(^e ^ 1)? we carry out similar steps used 



to arrive at (D.6) and exploit the symmetry of (|D.5|) to arrive at 

9b{w) 



- (fib{w - 2) + w^nl + /i^ j . 
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Hence, by assuming that fi^ < fia, we obtain upper and lower bounds on 



(D.3) and (D.4) in terms oi w = Ux + n^ with the aid of (D.7) as follows: 

(1 - ^aH)||ho||2 < llDholl^ < (1 + ^aH)||ho||2 • (D.8) 

It is important to realize that for some values of Ha, fJ-m, and w, the 



bounds in (|D.8|) are inferior to those obtained when ignoring the structure 

(D.9) 



of the concatenated dictionary D, i.e 

(1 - fidiw - l;;||iioll2 ^ 11^^^0112 ^ l-L -r H'd.yuj - ±;;||iio||2 
with \id = max{/ia, /!(,, fim}- However, for w > 2, ^^ = A^d, and 



hnllo < ||Dhol|2 < (1 + /id(wi - l))l|hc"' 



/^a ^ l-^m ~r 



llraW 




the RIP considering the structure of D in (D.8) turns out to be more tight 



than pX9 
than 



. For other values of w and/or /i^, (D.8) turns out to be less tight 



D.9|). In order to tighten the RIP in both cases, we consider 



|ho||2< ||Dho||2< (1 



(^w- 1 1 ho I 



2 ' 



where the coherence-based upper bound on the RIC of the concatenated 
dictionary D = [ A B ] corresponds to 



5,, 



mm 



- (^/ia(w - 2) + wv^/i^ + zi^j , ^id{w - 1) [ • 



Appendix D.2. Recovery guarantee 

We now bound the error ||h||2 and derive the recovery guarantee by fol- 
lowing the proof in Appendix A In the following, we only show the case 
where 

- [jiaiw - 2) + w^^l + ^il}j < fid{w - 1). 



The other case, i.e., where the standard RIP (D.9) is tighter than (D.8) 



readily follows from the proof in [Appendix A by replacing A by D, /i^ by 
/irf, and n^ by w. 
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Bounding the error on the signal support. We start by bounding the error 



^0ll2- 



Since /i^ < /^d, we arrive at 



.Ht-,H 



H-r\H-, 



h^D^Dho > h^'D^Dho - (h - ho)''D^Dh, 



> fl - 6.. 



u,J||"0|l2 



y"dWi||ho||2 - /idA/u;||ho||2 Cq 



c^l|ho|l2 -/^dV^llhollseo 



with 



d^l-6^- HdW = 1- —\^Ha + 2nd+ ^/fJ'l + ni,) + i^a- 
It is important to note that d is crucial for the recovery guarantee as it 



determines the condition for which BP-SEP in (13) enables stable separation. 
Specifically, if (i > or, equivalently, if 



w < 



2(1 + ;U„) 



m 



fia + 2fld + V^P^Tpl 

then the error on the signal support ||ho||2 is bounded from above as 



Iholls < 



e + r|)Vl + 6^, + i^dVwep 
d 



where cq = 2||w — wvv;||-^ with W = supp^(w). 

Bounding the recovery error. Analogously to the derivation in Appendix A. 3 
we now compute an upper bound on UhL, i.e.. 



|Dh||2 > (1 + /id)||h||2 - yUd||h||^ 



(D.IO) 



Finally, bounding UhlL similarly to Appendix A. 3 results in 



d + 2.JJI;jw^/l+J^ y/JI^{d + 2fidw) 

|h||2 < (£ + V) /, , . _, h eo- 



^/T+^d 
Cj{r] + e) + Csllw - wyvlli 



a/1 + ^^dd 



where the constants C-j and Cg depend on the parameters /Xa, /ib, /im, A*dj 
and w = Ux + Ue, which concludes the proof. 
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